✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

🔍 Preview links for changed docs

• docs/reference/api-reference/main.md

Just a general question/comment that came to my mind, do we want to have any limits on the number of SCPs that can be associated to a cluster ?
I was just trying to think what potential issues we might face when we have a large number of SCPs(if that is even a practical scenario) associated with a single ES cluster and if there is a practical maximum that we can enforce. One thing that came to my mind while thinking about scale was the annotation for soft owners, maybe we run into some kind of limit with the annotation map size? (I think the annotation limit is 256KB which I think should not be something to worry about ?)
Wondering if there are any other such things to consider.

buildkite test this -f p=gke,t=TestStackConfigPolicy*

buildkite test this -f p=gke,t=TestStackConfigPolicy*

@pkoutsovasilis wanted to follow up about my comment from earlier

#8917 (comment)

wondering if we should document what limits we might hit and maybe do some sort of scale testing to see if at a certain number it becomes problematic ? wdyt ?

@pkoutsovasilis wanted to follow up about my comment from earlier

#8917 (comment)

wondering if we should document what limits we might hit and maybe do some sort of scale testing to see if at a certain number it becomes problematic ? wdyt ?

+1 to that. I tested only with a trivial number of SCPs < 10. Maybe we should see if at higher cardinalities our reconciliation algorithm runs into problems. This could inform the documentation we write as a follow up and would allow us to give some guidance to users.

Just looking at the max annotation size for the soft-owners and assuming no other annotations exist, we could support probably ~ 2000 owners (63 char ns + 63 char name ~ 130 bytes + 32 bytes for the key). While this won't likely be the limitation I don't think we should advertise this as a limit. My gut feeling is that the number of SCPs should be < 100 and the ones targeting a single ES cluster < 10. But it would be good to have more than gut feel numbers 😄

Thanks for the follow-up @kvalliyurnatt! To be honest, as @pebrc already captured in his comment above, I'm not particularly worried about the owners annotations - those numbers are way beyond what we would ever advertise. Thinking about it, the limit that's more realistic to reach, I believe, is the stack config secret limit. We can easily hit that even when merging just 2 StackConfigPolicies that are quite lengthy, which could make us exceed the 1MB size limit of a secret. Everything else, except for the stack config secret size and the owners annotation, is pretty much the same, so this PR doesn't make any other limits easier to exceed.

So to accommodate for these my proposal is to limit the maximum number of SCPs that can target a given stack component to be 10 (it feels like a fairly high number to me). Also for the config policy secret size, we could check if the error coming from here is an IsRequestEntityTooLargeError, and if yes, set the status of the StackConfigPolicy accordingly. How does that sound @kvalliyurnatt @pebrc ?

I would not put a limit in code. What I was suggesting was to do a bit more "scale" testing. The overall size limit of the settings file is well known. This PR does not change anything in this regard. We are pretty confident that the annotation size is not a limitation in practice. What we do not know is how the controller behaves if there are very many SCPs that need to be reconciled on each and every change. How many conflicts will we see (if any). Any other issues we don't know of. So all I am arguing for is to test the limits of this implementation a bit before we give it to our users.